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Preface 



This report describes and evaluates the methods and procedures used for the field test of the 
Baccalaureate and Beyond Longitudinal Study (B&B:2000/01). The B&B:2000/01 field test included 
important changes from previous B&B surveys (conducted in 1994 and 1997) in its sample design and 
collection of data. For example, the current field test is the first to restrict the selection of institutions to 
those which are Title IV eligible. 

We hope that the information provided in this report will be useful to interested readers. The 
results of this field test were used to modify study procedures and instrumentation to be used in the full- 
scale B&B:2000/01, and should not be used to produce national estimates. Additional information 
about B&B:2000/01 and the B&B series is available on the web at http://nces.ed.gov/surveys/b&b. 
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Chapter 1 

Introduction, Background, and Purpose 



This document describes, summarizes, and evaluates the methodological procedures and 
results for the field test of the Baccalaureate and Beyond Longitudinal Study (B&B:2000/01). 

The Research Triangle Institute (RTI), assisted by MPR Associates, Inc., is conducting the 
B&B:2000/01 field test and subsequent full-scale study for the National Center for Education 
Statistics (NCES) of the U. S. Department of Education. 

This introductory chapter describes briefly the background, purposes, schedule, and 
products of the B&B:2000/01 study and the unique purposes of the field test. In Chapter 2, the 
field test design and method are described. Overall outcomes of data collection, as well as 
results of special studies, are presented in Chapter 3. Chapter 4 documents the quality of the data 
collected, and Chapter 5 summarizes the major recommendations for changes in design for the 
full-scale study. Materials used during the field-test survey are provided as appendices to the 
report and cited, where appropriate, in the text. 

A. Background and purpose of Baccalaureate and Beyond 

The purpose of the B&B follow-up study is to describe recent bachelor’s degree 
recipients and their activities in the year after graduation. B&B is designed to address issues 
such as time to degree, labor force participation and economic returns, participation in post- 
baccalaureate education, and student debt. 

As in the previous studies, B&B collects base-year data during the National 
Postsecondary Student Aid Study (NPSAS) data collection. For example, NPSAS:93 served as 
the base year for data collection for the first Baccalaureate and Beyond Longitudinal Study 
(B&B: 1993). These students were identified in NPSAS:93 as baccalaureate recipients during the 
1992-93 school year. The first follow-up took place one year after graduation (B&B:93/94) and 
the second occurred four years after graduation (B&B:93/97). As with B&B:93/94, the current 
study will collect follow-up data from students who were identified as baccalaureate recipients in 
the NPSAS:2000 survey, one year after graduation. And, as with B&B:93/94 and the prior 
NCES Recent College Graduates (RCG) study series, it will allow study of graduates’ 
experiences as undergraduates and their initial forays into graduate education and the labor 
market. It also offers an important opportunity to study early outcomes of newly qualified 
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teachers, including entry and attrition, certification, and participation in the teaching profession, 
and enables the continuation of a series of earlier NCES reports. 

One of the major issues addressed in the B&B Longitudinal Study is how long it takes 
undergraduates to earn a bachelor’s degree. B&B:2000/01 will permit continuation of the “time 
to degree” trend data provided by the preceding RCG cross-sectional and B&B: 93 longitudinal 
surveys, which are periodically reported as indicators in the annual Condition of Education 
published by NCES. 

B&B also provides information about the economic returns to bachelor’s degree 
completion. B&B:2000/01 extends trend data from RCG and B&B:93/94 on employment status, 
income, and unemployment one year after graduation. In addition, B&B provides information 
about the supply of graduates trained in science and technology, and about the graduates who are 
employed in those fields. The issue of student debt, through both the federal student loan 
programs and the increased use of credit cards, is particularly relevant for recent college 
graduates. Data collected for B&B provide an estimate of the average debt accumulated by 
students in order to complete a bachelor’s degree, and how this may affect future plans. With the 
addition of federal student loan history data from National Student Load Data System (NSLDS), 
B&B:2000/01 will be able to expand and improve information on student debt and repayment. 

B&B:93/94 provided the data for the NCES report on newly qualified teachers. 
B&B:2000/01 offers the opportunity for comparative analyses to determine whether recent 
graduates’ perspectives regarding teaching or tendencies to enter teaching changed in the seven 
years between these surveys. 

B. Overall schedule and products of B&B:2000/01 

B&B:2000/01 full-scale data collection is scheduled for June through December 2001. 
The operational schedule for the B&B:2000/2001 field test is presented in table 1.1. 



Table 1.1 - Start and end dates for major B&B:2000/2001 field test activities 



Activity 


Start date 1 


End date 2 


Sampling 


12/6/99 


12/29/99 


Preload base-year data into computer-assisted telephone 


3/17/00 


3/17/00 


interviewing (CATI) records 






CATI training 


3/20/00 


3/25/00 


Mailed student prenotification letters 


3/27/00 


3/27/00 


CATI data collection 


3/28/00 


7/2/00 



'This is the date on which the activity was initiated for the first applicable school and/or its associated students. 
2 This is the date on which the activity was completed for the last applicable school and/or its associated students. 



The following publications and data products will be produced: 

• electronically documented, restricted access research files (with associated electronic 
codebooks) for research data users; 
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• a Data Analysis System (DAS) for public access to B&B:2000/01 data; 

• methodology reports for both the field test and the full-scale studies, providing 
details of sample design and selection procedures, data collection procedures, 
weighting methodologies, estimation procedures and design effects, and the results of 
nonresponse bias analyses; and 

• two descriptive summaries: (1) a B&B descriptive summary with an essay, and (2) a 
report on newly qualified teachers. 

C. Purpose of the field test 

The major purpose of the B&B:2000/01 field test was to plan, implement, and evaluate 
all operational and methodological procedures, instruments, and systems proposed for use in the 
full-scale study. Many such methodological features, representing enhancements or refinements 
to previously used B&B approaches, had not been fully tested in the past. Using and testing 
methodologies in the field test that parallel the data collection procedures proposed for the main 
B&B data collection allow such procedures to be modified as necessary, before the much larger 
(and more expensive) full-scale data collection activities begin. 
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A. Sample design 

The B&B:2000/01 field test sample is a sample of postsecondary students in the United 
States and Puerto Rico who completed a baccalaureate degree between July 1, 1998, and June 30, 
1 999. Students were identified as potentially eligible for the B&B field test if the institution or 
student indicated that the student had received or was a candidate to receive a baccalaureate 
degree between July 1, 1998, and June 30, 1999. All potentially eligible B&B students who 
responded in the NPSAS:2000 field test were sampled for the B&B follow-up survey. A 
subsample was selected from the potentially eligible B&B students who did not respond in the 
NPSAS:2000 field test. 

1. NPSAS institutional sample 

Effectively, all U.S. institutions eligible for Title IV aid 1 that offered academically or 
vocationally oriented postsecondary programs were eligible for NPSAS:2000. 2 Specifically, to 
be eligible for NPSAS:2000, anon-military-academy educational institution must 

• offer an educational program designed for persons who have completed secondary 
education; 

• offer more than just correspondence courses; 

• offer at least one academic, occupational, or vocational program of study lasting at 
least 3 months or 300 clock hours; 

• offer courses that are open to more than the employees or members of the company or 
group (e.g., union) that administers the institution; 

• be located in the 50 states, the District of Columbia, or Puerto Rico; and 



1 U.S. military academies were excluded due to their atypical funding/tuition base. 

2 The NPSAS universe for the field test included all eligible institutions in the 1997-98 Integrated Postsecondary 
Education Data System (IPEDS) Institutional Characteristics (IC) file. 
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• have a signed Title IV participation agreement with the Department of Education. 

Institutions providing only vocational, recreational, remedial, or correspondence courses, or only in- 
house courses for their own employees, were excluded. 

Institutions selected for the NPSAS:2000 field test were not selected for the full-scale study 
because participation in both surveys was considered excessively burdensome. To accomplish this, 
the field test sample was selected after institutions large enough to be certainty institutions in the 
full-scale survey were deleted from the field test institutional sampling frame. Then, a stratified 
simple random sample of institutions was selected for the field test, using the same 22 strata as the 
full-scale study. Although no probability-based inferences were planned for the field test, a 
probability-based sample was used because the complement of the field test sample will be used for 
the full-scale study-sampling frame. 

An important benefit of this method of selecting the institutions for the field test is that a 
more up-to-date institutional sampling frame could be constructed for the full-scale survey without 
loss of the ability to generalize to the full population. The full-scale sampling frame will be 
constructed from the 1 998-99 IPEDS IC file, which became available after the field test sample had 
been selected. Institutions that had been selected for the field test sample will be deleted from the 
full-scale sampling frame so that they will not be selected for the full-scale sample. The probability 
of selection for the full-scale study will be adjusted for institutions on the sampling frame based on 
the probability that they were not selected for the field test sample. 

Nearly twice as many institutions as needed were selected in the simple random sample for 
the field test so that the field test sample could be selected purposively from this sample. Three 
institutions in Puerto Rico were purposively selected to evaluate the viability of alternative methods 
of locating and interviewing students located there, and to check on whether the improved response 
rates, which RTI achieved in Puerto Rico in NPSAS:96, would continue. Clusters of institutions 
were selected in several cities to provide an adequate number of students for testing the field 
interviewing procedures. The remaining field test institutions were selected to represent the 22 
institutional strata. 

In total, 74 institutions were selected for the field test with the expectation that this sample 
size would yield 66 institutions that both were eligible and would provide lists for student sampling. 
A breakdown of sampled institutions by original institutional stratum is provided in table 2.1 . This 
table also shows, in total and by institutional stratum, eligibility rates and rates for providing student 
lists. Overall, 98 percent of the sampled institutions met NPSAS eligibility requirements, and of 
those, about 86 percent provided lists for student sampling. 
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Table 2.1 — NPSAS:2000 field test institutional sampling, eligibility, and list providing, by 
sampling stratum 



Institution sampling stratum 


Sampled institutions 


Eligible institutions 


Provided lists 


Number 


Percent 1 


Number 


Percent 2 


Number 


Percent 3 


Total 


74 


100.0 


73 


98.6 


63 4 


86.3 


Public 














1 Less-than-2-year 


3 


4.1 


3 


100.0 


3 


100.0 


2 2-year 


2 


2.7 


2 


100.0 


2 


100.0 


Total less-than-4-year 


5 


6.8 


5 


100.0 


5 


100.0 


3 Bachelor’s, high ed 5 


2 


2.7 


2 


100.0 


2 


100.0 


4 Bachelor’s low ed 6 


4 


5.4 


4 


100.0 


4 


100.0 


5 Master’s, high ed 


4 


5.4 


4 


100.0 


3 


75.0 


6 Master’s, low ed 


4 


5.4 


4 


100.0 


4 


100.0 


Total 4 -year non-doctorate-granting 


14 


18.9 


14 


100.0 


13 


92.9 


1 Doctorate-granting, high ed 


2 


2.7 


2 


100.0 


2 


100.0 


8 Doctorate-granting, low ed 


4 


5.4 


4 


100.0 


2 


50.0 


9 First-professional -granting, high ed 


2 


2.7 


2 


100.0 


2 


100.0 


10 First-professional-granting, low ed 


6 


8.1 


5 


83.3 


5 


100.0 


Total 4-year doctorate-granting 


14 


18.9 


13 


92.9 


11 


84.6 


Private, not-for-profit 














1 1 Less-than-2-year 


2 


2.7 


2 


100.0 


2 


100.0 


1 2 2 -year 


2 


2.7 


2 


100.0 


1 


50.0 


Total less-than-4-year 


4 


5.4 


4 


100.0 


3 


75.0 


1 3 Bachelor’s, high ed 


2 


2.7 


2 


100.0 


2 


100.0 


14 Bachelor’s, low ed 


5 


6.8 


5 


100.0 


5 


100.0 


15 Master’s, high ed 


2 


2.7 


2 


100.0 


2 


100.0 


16 Master’s, low ed 


5 


6.8 


5 


100.0 


4 


80.0 


Total 4 -year, non-doctorate-granting 


14 


18.9 


14 


100.0 


13 


92.9 


1 7 Doctorate-granting, high ed 


2 


2.7 


2 


100.0 


1 


50.0 


1 8 Doctorate-granting, low ed 


5 


6.8 


5 


100.0 


4 7 


80.0 


19 First-professional-granting, high ed 


2 


2.7 


2 


100.0 


1 


50.0 


20 First-professional -granting, low ed 


7 


9.5 


7 


100.0 


6 7 


85.7 


Total 4-year, doctorate-granting 


16 


21.6 


16 


100.0 


12 4 


75.0 


Private, for-profit 














21 Less-than-2-year 


4 


5.4 


4 


100.0 


3 


75.0 


22 2 -year or more 


3 


4.1 


3 


100.0 


3 


100.0 


Total private , for-profit 


7 


9.5 


7 


100.0 


6 


85.7 



Percent is based on overall total within column. 

2 Percent is based on number sampled within row. 

3 Percent is based on number eligible within row. 

includes two institutions which agreed to provide lists but did not do so in the time provided 

5 A school was classified as “high ed” if it was in the top 20 percent of its stratum in terms of the numbers of baccalaureate students 
graduating with education degrees. 

6 A school was classified as “low ed” if it was not in the top 20 percent of its stratum in terms of the numbers of baccalaureate 
students graduating with education degrees. 

includes one institution which agreed to provide lists but did not do so in the time provided. 

NOTE: Most first-professional-granting institutions award doctor’s degrees as well as first-professional degrees. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Postsecondary Student Aid Study:2000 
field test. 
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2. Design and Method of the Field Test 



2. NPSAS student sample 

Not all students enrolled in eligible institutions were considered eligible for NPSAS. In 
addition to being enrolled at a NPSAS-eligible institution during the appropriate time frame (for the 
field test, between July 1, 1998, and April 30, 1999; for the full-scale study, between July 1, 1999, 
and June 30, 2000), NPSAS-eligible students had to be: enrolled in either (1) an academic program; 
(2) at least one course for credit that could be applied toward fulfilling the requirements for an 
academic degree; or (3) an occupational or vocational program that required at least 3 months or 
300 clock hours of instruction to receive a degree, certificate, or other formal award. Simul- 
taneously, they could not be concurrently enrolled in high school, nor enrolled solely in a GED or 
other high school completion program. 

Students who received a baccalaureate degree at any time between the appropriate dates for 
the field test (between July 1, 1998, and June 30, 1999) were eligible for the NPSAS and the 
Baccalaureate and Beyond studies. 

Students were selected from “unduplicated” 3 student lists provided by participating 
institutions, using the same procedures to be implemented in the full-scale study. While schools 
were made aware of student eligibility requirements, as in previous waves of NPSAS, the bulk of 
the student eligibility determination was accomplished after sampling from the provided lists (i.e., 
during record abstraction or student interviewing). Incorrect information provided by institutions as 
to student status resulted in some other misclassification errors, which were also corrected after 
sampling. 

Students were stratified within selected institutions into seven strata. Separate strata were 
established for baccalaureate degree completers, undergraduates, first-professional students, and 
other graduate students. The baccalaureate stratum was subdivided into two mutually exclusive 
strata based on whether students’ major field of study was business or another field. Three graduate 
strata were defined as: students in master’s degree programs, students in doctorate degree programs, 
and other graduate students. Stratum sampling rates were predetermined for each institution to yield 
the desired stratum sample sizes and minimum institution sample sizes. 

Business baccalaureate recipients were sampled at lower sampling rates than other 
baccalaureate recipients because large proportions of all baccalaureate degrees are awarded to 
business majors. Differential sampling rates were also used for the three types of graduate students 
in order to get adequate representation of students pursuing doctoral degrees and to limit the sample 
size for “other” graduate students, who are of limited inferential interest. Established sampling rates 
were applied to the unduplicated student lists to attain the sample using stratified systematic 
sampling procedures. The sample was constrained so that (1) no less than 25 students were to be 
selected from each institution, even if the sampling rate had to be raised; and (2) the total sample 



3 In some instances, the lists could be unduplicated by the supplying institutions. However, in many cases, institutions 
were unable (or unwilling) to supply unduplicated lists, and the unduplicating process was accomplished by contractor 
staff. 
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2. Design and Method of the Field Test 



from an institution did not exceed the expected sample size based on the 1997-98 IPEDS 
information by more than 50, even if the rates had to be reduced. The sample size was monitored by 
strata and sampling rates were adjusted, where appropriate. 

The achieved field test student sample sizes are shown in table 2.2 by institutional type and 
student stratum. About half of the overall sample, more than half of the baccalaureate sample, and 
almost half of the other undergraduate sample were selected from public institutions (reflecting the 
higher undergraduate enrollment in such institutions); however, the graduate/first-professional 
sample had a slightly higher percentage selected from private, not-for-profit institutions than from 
public institutions. During the full-scale study, the sample sizes in each student stratum will be 
closely monitored and the sampling rates adjusted, as necessary, to achieve target sample sizes. 

Table 2.3 shows the base-year NPSAS field test response rates for the B&B cohort. A total 
of 1,302 potential baccalaureate degree recipients were identified using institutionally provided lists 
of students who graduated or were candidates to graduate between July 1, 1998, and June 30, 1999. 
Of the 1,302 potential baccalaureate degree recipients sampled during the base year, 196 were from 
institutions that submitted data-file CADE. The collection of CADE information via data file was a 
procedural test, and these cases were not intended to be loaded into the computer-assisted telephone 
interviewing (CATI) system because there would not be sufficient time to work these cases in CATI 
during the base year field test. Therefore, the 196 data file cases were excluded from both the base 
year and follow-up samples. Additionally, 61 of the 1,302 B&B sampled students were 
subsequently determined in the NPSAS field test survey to be ineligible. Of the remaining 1 ,045 
students sampled as B&B who were eligible for NPSAS, 797 (76.3 percent) participated in NPSAS. 
Students in public institutions had the highest NPSAS response rates for the B&B cohort (80.2 
percent). 

The field test student sample was selected to represent the various institutional sectors so 
that we could properly test the study instruments and operations. However, since we are not 
interested in making statistical inferences from field test data, ensuring population coverage was not 
as critical for the field test design as it will be in the full scale study. 
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Table 2.2 — Initial classification of NPSAS:2000 field test student sample, by school type and student stratum 
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2. Design and Method of the Field Test 



Table 2.3 — B&B cohort NPSAS:2000 field test response rates, by institution type 



Institution type 


Eligible sample 
students 


Participating 

students 


Response rate (%) 


All institutions 


1,045 


191* 


76.3 


Institutional level 








Bachelor’s or less 


314 


243 


77.4 


Master’s 


292 


217 


74.3 


Doctor’s 


115 


84 


73.0 


First-professional 


324 


253 


78.1 


Institutional control 








Public 


565 


453 


80.2 


Private, not-for-profit 


480 


344 


71.7 


Private, for-profit 


0 


0 


(t) 


Institutional sector 








Public, bachelor’s or less 


120 


98 


81.7 


Public, master’s 


172 


136 


79.1 


Public, doctor’s 


76 


62 


81.6 


Public, first-professional 


197 


157 


79.7 


Private, not-for-profit, bachelors or less 


194 


145 


74.7 


Private, not-for-profit, master’s 


120 


81 


67.5 


Private, not-for-profit, doctor’s 


39 


22 


56.4 


Private, not-for-profit, first-professional 


127 


96 


75.6 


Private, for-profit 


0 


0 


(t) 



t Not applicable. 

includes 79 NPSAS-eligible students who did not receive a baccalaureate degree during the NPSAS year (i.e., false positives) 
and excludes 12 responding students who were not in the B&B sample (i.e., false negatives). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Postsecondary Student Aid 
Study:2000 field test. 



3. B&B follow-up student sample 

The sampling frame for the B&B:2000/01 field test was constructed by considering the 
following types of students from the NPSAS: 2000 field test: 

• students who were located and interviewed during the NPSAS:2000 field test, and 
confirmed to be baccalaureate recipients between July 1, 1998, and June 30, 1999; 

• students who were sampled as members of the B&B cohort, and located but not 
interviewed in the NPSAS:2000 field test; 

• students who were sampled as baccalaureate recipients, but not located for the 
NPSAS:2000 field test; 
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2. Design and Method of the Field Test 



• students who were sampled as baccalaureate recipients but classified as 
exclusions 4 forNPSAS; and 

• students who were sampled as baccalaureate recipients but did not meet the 
NPSAS eligibility requirements specified in section A. 2. 

For each of the above categories, table 2.4 shows the distribution of NPSAS: 2000 field test and 
the B&B follow-up sample. 

The first three types of students listed above formed the three sampling strata for the 
B&B:2000/01 field test. All students were selected from the first stratum. Within the other two 
strata, the students were sorted by whether or not they were included in the NPSAS incentive 
experiment, 5 and then they were further sorted by the nine-level institutional sector. Within each 
stratum, a systematic sample was selected from the sorted frame, which ensured proportional 
representation of the students within strata by whether or not they were included in the incentive 
experiment and by sector. The total B&B follow-up field test sample size consisted of 
855 students, 672 of whom were not in the NPSAS incentive experiment, and 1 83 of whom were 
in the experiment. The B&B follow-up sample distribution by institutional sector is shown in 
table 2.5. None of the exclusions or ineligible students was sampled. 

B. Data collection design 

1. Locating 

The basic B&B:2000/01 design involved tracing sample members to their current location 
and conducting a computer-assisted telephone interview with them about their experiences since 
the NPSAS:2000 field test interview approximately one year earlier. The data collection 
activities, including locating, are shown in figure 2.1. While the flow shown is sequential for any 
given case, these activities were quite dynamic. At any given time during the 
locating/interviewing period, different sample members were at markedly different stages in the 
flow. 



4 Students who had died or were incarcerated, institutionalized, or out of the country for the duration of the data 
collection period were classified as exclusions for NPSAS. 

5 See chapter 2, section H.2, for more details. 
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Table 2.4 — Distribution of NPSAS:2000 field test and B&B:2000/01 follow-up sample sizes 



Disposition of NPSAS:2000 field test B&B cases 


NPSAS:2000 B&B 
field test sample 


B&B:2000/01 follow-up 
field test sample size 


Total 


1,034 


855 


NPSAS:2000 respondent, confirmed B&B 


730 


730* 


Located but not interviewed in NPSAS :2000 


117 


59 


Not located 


131 


66 


Exclusion 


21 


0 


Confirmed ineligible during NPSAS :2000 


35 


0 



*Includes 718 respondents from the B&B base-year sample and 12 respondents who were not initially selected as part of the B&B 
sample but were determined to be B&B eligible during the base year (i.e., false negatives). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study: 2000/01 field test. 



Table 2.5 — B&B:2000/01 field test sample sizes, by sector of NPSAS institution 



NPSAS:2000 institutional sector 


B&B: 2000/01 field 
test sample size 


Total* 


855 


Public, 4-year, non-doctorate granting 


250 


Public, 4-year, doctorate-granting 


232 


Private, not-for-profit, 4-year, non-doctorate-granting 


249 


Private, not-for-profit, 4-year, doctorate-granting 


123 



*The total includes one case that was sampled at a 2-year institution but was determined to be eligible for B&B at another 
institution. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 
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Figure 2.1 — Flow of locating and interviewing activities for B&B:2000/01 field test 
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2. Design and Method of the Field Test 



a. Pre-CATI locating 

Locating information obtained during the NPSAS:2000 field test was incorporated into the 
B&B:2000/01 field test locator database, and sent in batch mode to the U.S. Postal Service National 
Change of Address (NCOA) system and Telematch in February of 2000. These services provided 
updated address and telephone number information respectively. Following the first round of 
NCOA and Telematch batch processes and after updating of the locator database with the new 
information, a student mailing was sent to all sample members one week before data collection 
started. The purpose of the mailing was to inform them of the study and their rights as participants. 
In addition, the student mailing gave sample members the opportunity to complete and return an 
address update sheet. Each sample member received a lead letter, address update sheet, information 
leaflet, and business reply envelope (see appendix B). All locating information obtained from the 
student mailing was entered into the locator database. 

b. CATI locating 

Locating and tracing activities took place concurrently with efforts to gain cooperation from 
and interview sample members. When assigned a case, the telephone interviewer called the 
telephone number designated by the system as the best number (i.e., the number among all available 
locator numbers that appeared to have the greatest potential for contacting the sample member) and 
attempted to interview the designated sample member. When the person answering the call said that 
the sample member could not be reached at that number, the interviewer asked the person how to 
contact the sample member. If this query did not provide the information needed, the interviewer 
initiated tracing procedures, using all information available to call other contact persons in an 
attempt to locate the student. When all tracing options available to the interviewer were exhausted 
without success, the case was assigned to RTFs Tracing Operations Unit (TOPS) for intensive 
tracing. 



c. Intensive tracing efforts 

TOPS had access to both proprietary and public-domain data. It had real-time access to 
several consumer databases, which contained current address and phone listings for the majority of 
consumers with a credit history. In addition to the propriety databases, TOPS had access to various 
other information sources, such as data miners, commercial list houses, and NCOA via leased line. 
These sources provided the following searches: name, address, neighbor, business, phone matching 
searches, and status as decedent, incarcerated, incapacitated, or military personnel. TOPS employed 
these various information sources to locate respondents. 

A two-tiered intensive-tracing plan was used to locate B&B sample members. The first tier 
involved identifying sample members with social security numbers (SSNs) and processing that 
information through consumer database searches. If a search generated a new telephone number, 
that case was sent back to CATI for telephone interviewing. If a new address was generated, but no 
telephone number, tracers called directory assistance or accessed other databases to obtain telephone 
numbers for CATI. This first level of effort minimized the time that cases were out of production. 
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All remaining cases (those lacking new information from the SSN search) underwent a more 
intensive level of tracing in the second tier. This approach involved the following procedures: 

(1) checking directory assistance for telephone listings at various addresses; (2) using electronic 
reverse-match databases to obtain the names and telephone numbers of neighbors and then calling 
the neighbors; (3) calling persons with the same unusual surname in small towns or rural areas to 
see if they were related to or knew the sample member; (4) contacting the current or last known 
residential sources such as the neighbors, landlords, current residents, tax assessors, realtors, and 
other business establishments related to previous addresses associated with the sample member; (5) 
calling colleges, military establishments, and correctional facilities to follow up on leads generated 
from other sources; and (6) checking various tracing Web sites. Tracers checked new leads 
produced by these tracing steps to confirm the addresses and telephone numbers for the sample 
members. When the information was confirmed, that case was returned to CATI for telephone 
interviewing. If the information could not be confirmed (e.g., there were no working telephone 
numbers or numbers for relevant neighborhood sources were unpublished), the case was reviewed 
by team leaders in TOPS and the RTI Telephone Survey Unit. 

C. Data files 

The final B&B:2000/01 field-test data file will be prepared in accordance withNCES- 
specified Electronic Codebook (ECB) format guidelines. Activities important to ensuring quality 
across data file preparation methods are described below. 

1. Data availability throughout data collection period 

During data collection, RTI consistently updated master data files containing completed case 
data. Batch processes included methods for extracting raw data from Computer-Assisted Survey 
Execution System (CASES) interview data files. A master data file structure was retained across all 
CATI applications in order to concatenate data into a single set of data containing all respondents. 
CATI data were extracted periodically to facilitate advance preparation of variable documentation. 
This also enabled review of frequency distributions and data analysis while data were still being 
collected. 



2. Data editing 

Data became available for editing during the data collection period, which allowed for 
feedback to the field on data quality and more accurate analysis of response data. As a quality 
check, the original skip logic was recreated to ensure that respondents followed the appropriate path 
within the CASES instruments. These edit checks were important for correcting errant paths that 
the interviewer may have followed but later corrected. Edit checks also ensured that for particular 
data elements, responses occurred logically. This process also allowed evaluation of the other case: 
when questions the respondent should have been asked were missed. Reserve codes indicated 
instances where raw data were updated to reflect the proper logical path. During data collection, 
interviewing staff was able to notify project staff of CATI irregularities via “problem sheets” so that 
problems in instrument logic could be corrected. 
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D. Instrument design 

The B&B:2000/01 field test student telephone interviews were conducted using CATI 
technology. In preparation for the development of the CATI instrument, a comprehensive set of 
data elements was developed from a thorough review of the data elements used for the B&B:93 
cohort, their relationship to the NPSAS:2000 data elements, and their relevance to current research 
and policy issues. A preliminary set of B&B:2000/01 data elements was refined with input from the 
study’s Technical Review Panel (see appendix A for a list of members) as well as from NCES and 
other Department of Education staff. The final set of data elements, presented in appendix C, was 
approved by the Office of Management and Budget (OMB) before data collection started. 

Based on the set of data elements, the CATI instrument was structured by identifying section 
topics and determining the progression of items within sections. Individual items were designed 
with several goals in mind: (1) using NPSAS:2000 and B&B:93/94 items when feasible; (2) 
ensuring consistency withNPSAS:2000 and B&B:93/94 items when items were not identical; and 
(3) identifying and preparing wording for item verifications and probes as necessary. Facsimile 
instruments are provided in appendix D. 

Instrument sections were reviewed on a flow basis by NCES and by selected contractor and 
subcontractor staff. As depicted in figure 2.2, the first section determined eligibility for sample 
members who did not participate in NPSAS:2000. The following sections collected information 
pertaining to postsecondary enrollment since high school completion, respondent demographics, 
post-baccalaureate education and employment, and experiences with teaching. 

To minimize the interview burden on respondents, the CATI instrument used existing data 
whenever feasible. Base-year data from the NPSAS:2000 field test interview were preloaded into 
the CATI interview; this dictated the flow of many portions of the interview. Certain questions 
were asked only if the data were missing from the prior interview. 

The CATI interviews were programmed using CASES 4.3 software. The CATI system 
presented interviewers with screens of questions to be asked of the respondents, with the software 
guiding the interviewer and respondent through the interview, automatically skipping inapplicable 
questions based on prior response patterns. Wording for probing and verification was suggested 
when a respondent provided a response that was out of range for a given item. As the CATI 
instrument was being designed and programmed, instrument documentation was entered into an 
integrated data dictionary system (DDS), which subsequently enabled users to produce deliverable 
data files with CATI variable documentation. An abbreviated instrument was developed for the 
purpose of interviewing special respondent groups such as sample members whose primary 
language was Spanish. The facsimile abbreviated instrument, presented in appendix D, focused on 
the respondent’s post-baccalaureate enrollment and work experiences. 
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Figure 2.2 — Structure and flow of B&B:2000/01 follow-up field test student CATI 




Section A: Eligibility 
Determination 

Enrollment in NPSAS school; other 
school enrollment; candidacy for 
bachelor's degree 



Section B: Undergraduate 
Enrollment 

Enrollment in NPSAS year; prior 
enrollment; program of study; 
financial aid; employment while 
enrolled 



Section D: Student Background 

Marital status; family composition; 
spouse's education; military 
service; community service 
participation; financial obligations 



Section E: Post-baccalaureate 
Education 

Graduate school/other enrollment; 
financial aid; employment while 
enrolled; plans for enrollment 



Section F: Post-enrollment 
Employment 

Employment status; type of 
occupation/industry; employer size; 

salary and benefits; On-the-job 
training; licensure; unemployment 



Section G: Teaching 
Experiences 

Type of position; location; 
certification; preparation and 
training; school support; plans to 
teach in the future 



NOTE: All the original Section C items were moved to other sections. To avoid introducing confusion into the CATI 
programming, however, the remaining sections have not been relettered. 
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Once all CATI sections had been programmed, test cases were developed and preloaded 
for testing the instrument and for training telephone and field interviewers. Project staff and staff 
from NCES systematically tested the CATI instrument before the interviewer training. Finally, 
preload files containing data from NPSAS:2000 and the Department of Education databases were 
prepared and loaded into the CATI system to both guide the interview and assist sample member 
locating efforts. Data collection ensued only after all these tasks were complete. 

E. Training of interviewers 

The field test training program was designed to maximize the trainees’ active 
participation. Training manuals included a training guide, an interviewer’s manual, and a 
question-by-question specification manual. Training for telephone interviewers and supervisors 
was conducted in March 2000 and consisted of lectures, demonstrations, and hands-on practice 
exercises with the instrument and online coding modules. Trainees were introduced to the 
procedural aspects of conducting B&B: 2000/01 and were given a thorough review of the 
questionnaire. Interviewers were also trained in techniques for gaining cooperation with sample 
members, parents, and other contacts, as well as techniques for addressing the concerns of 
reluctant participants and avoiding refusals. A copy of the training agenda and the table of 
contents from the training manual are located in appendix E. 

E Telephone interviewing 

CATI locating and interviewing were conducted from March 28, 2000, through July 2, 
2000. CATI procedures included attempts to locate, to gain cooperation from, and to interview 
study sample members by telephone. For NPSAS:2000 field test nonrespondents, NPSAS and 
B&B eligibility determination was also necessary. A reliability reinterview consisting of a subset 
of items from the full instrument was conducted for a subsample of respondents (79). 

Locating information gleaned from the pre-CATI locating sources described above was 
preloaded for each case. Additionally, information previously collected through NPSAS:2000 
was preloaded to personalize interviews and to reduce respondent burden. 

An automated call-scheduler assigned cases in the CATI sample to interviewers based on 
time of day, day of week, existence of precise appointments, and type of case. Scheduler case 
assignment was designed to maximize the likelihood of contacting and interviewing sample 
members. Cases were assigned to various queues for this purpose. Some of the queues included 
new cases, Spanish-language cases, initial refusals, and various appointment queues (firm 
appointments set by the sample member, appointments suggested by locator sources, and 
appointments for cases which were initial refusals). 

For each case, a calling roster determined the names and telephone numbers for the 
interviewers to call. The roster included school-provided and/or student-provided address 
information (student permanent, student local, parent, and other contacts) from the NPSAS :2000 
field test. Up to six roster-lines were preloaded with contact information. New roster-lines were 
added as necessary during the field test as the result of CATI tracing and intensive tracing efforts. 
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Once located, some cases required special treatment. To gain cooperation from those 
who initially refused to participate (including locator sources who acted as “gatekeepers,” 
preventing access to the sample member), certain interviewers were trained in refusal conversion 
techniques. Sample members and their locator sources who spoke only Spanish, primarily 
located in Puerto Rico, were initially assigned to bilingual CATI interviewers. 

G. Integrated management system (IMS) 

All aspects of the study were under the control of an integrated management system 
(IMS) which consisted of several components, or modules:: 

• Management 

• Receipt Control System (RCS) 

• CATI 

• Field Case Management System (FCMS) 

• Assignment/Transfer System 

This modular structure allowed for the streamlining of related tasks and resulted in a centralized, 
easily accessible repository for project data and documents. 

The Management module of the IMS contained tools and strategies to assist the project 
staff and the NCES project officer in managing the study. All information pertinent to the study 
could be found here via the World Wide Web in a secure desktop environment: schedules, 
monthly progress reports, project plans and specifications, information related to the Technical 
Review Panel (TRP), and project deliverables. Also available in the management module was 
the latest version of the CATI instrument for testing and review, daily Receipt Control System 
(RCS) module status reports, and daily data collection reports. 

The RCS module monitored activities related to data collection, including tracing and 
locating, thereby enabling project staff to perform stage-specific activities, track case status 
closely, identify problems early, and implement solutions effectively. Several applications used 
the RCS’s locator data for daily tasks: The mailout program produced mailings to parent/contacts 
and sample members, the query system enabled administrators to review the locator information 
and status for a particular case, and the mail return system enabled project staff to update the 
locator database as mailings or reply sheets were returned or forwarded. The RCS also interacted 
with the Tracing operation system database, sending locator data between the two systems as 
necessary. 

The CATI module managed development of the CATI instrument within the DDS. The 
DDS consisted of a set of linked relational files and associated utilities for developing and 
documenting the instrument. Developing the CATI instrument with the DDS ensured that all 
variables were linked to their item/screen wording and were thoroughly documented. Also 
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included within the CATI module was online coding software (“user exits”) that collected detail 
on schools attended, enrollment, industry, occupation, and field of study. 

The Field Case Management System (FCMS) module facilitated activities performed by 
the field interviewers. The FCMS allowed field staff to conduct tracing and personal 
interviewing activities, to communicate with RTI staff via electronic mail, to transmit completed 
cases, to receive new cases, and to transmit production time and expense (PT&E) data back to 
RTI. 



The Web-based Assignment/Transfer System enabled the field supervisor to make all 
case assignments to field interviewers as well as to track progress of cases being worked in the 
field. 

H. Methodological experiments and evaluation approaches 

1. Purpose of evaluation procedures 

Evaluating field test procedures can lead to improved procedures for the subsequent full- 
scale study. Each major component of the field test was evaluated. The methodology consisted 
of both formative and summative evaluations. Formative evaluations were of an ongoing nature, 
designed to assess tasks at intermediate stages so that the effects of employing alternate 
methodologies could be analyzed and modifications could be made and assessed before the 
affected task was completed. Summative evaluations will be used to optimize procedures in the 
full-scale study. A summary of B&B:2000/01 field test evaluations that were planned and 
implemented is provided in table 2.6. 

Various measures were employed to assess the quality of data collection, including 
quality assurance (or quantitative) monitoring, qualitative monitoring, and quality circle 
meetings. The primary objective was to pinpoint any problems with the questionnaire and 
correct them early in data collection. Quality assurance monitoring assessed the quality of the 
telephone interviewing, with respect to question delivery and coding of responses. It is explained 
in greater detail in chapter 4. Qualitative monitoring evaluated whether interviewing procedures 
were implemented as intended and were effective. The utility of the interview items was also 
assessed. On occasion, monitoring revealed the need for individual interviewer retraining (e.g., 
better explanation of the nature of the study, or techniques for refusal avoidance) which was 
conducted immediately. Weekly quality circle meetings — during which interviewers, 
supervisors, and project technical staff met to discuss data collection issues — proved valuable in 
identifying problems with the instrument as well as in building rapport among interviewers and 
technical staff. Summaries of the meetings were rapidly disseminated to all interviewers and 
supervisors so that those who were unable to attend also benefited. 

The study design included a component for direct evaluation of data quality. A reliability 
reinterview was administered to a randomly selected subsample of field test respondents to assess 
the short-term stability of selected items. The results of the reinterview analysis are presented in 
chapter 4. 
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Table 2.6 — Summary of planned B&B:2000/01 field test evaluations 



'Major aj||||j evaluation | 


i w ' 


Evaluation approaches 


CATI instrument development 


• 


Debrief instrument “testers/debuggers” to determine 
appropriateness and accuracy of instrument flow/logic. 


Training 


• 


Debrief CATI supervisors and interviewers. Debrief field 
supervisors and field locators/interviewers. 


Centralized tracing activities 


• 


Debrief tracing staff and supervisors. Analyze all levels of tracing 
results and costs. 




• 


Analyze outcomes and costs/benefits of CATI-extemal tracing 
activities. 


CATI administration/ data quality 


• 


Analyze silent monitoring quality control data. Analyze CATI 
operational parameters (e.g., numbers of calls per case, total 
interviewer hours per completed interview). 




• 


Debrief interviewers, refusal converters, bilingual interviewers, 
monitors, and supervisors. 




• 


Analyze administration time statistics, overall, within section, and 
for individual questions and blocks of questions. 




• 


Analyze rates of interview nonresponse, early and subsequent 
break-off, types of response inconsistencies detected during 
interview administration, and nonresponse patterns. 




• 


Analyze effect of prior receipt of financial incentive on response 
rates. 




• 


Analyze response reliability of reinterviews for selected items 
(sub sample). 




• 


Assess feedback from mock interviews conducted with TRP 
members. 




• 


Compare estimates for selected variables between CATI 
respondents and nonrespondents. 


Online coding 


• 


Analyze success/accuracy of online coding of IPEDS code, 
industry/occupation, and major course of study. 


File development 


• 


Observe and document any procedural difficulties encountered in 
preparation of raw CATI files. 
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2. Incentive experiment 

The field test included a methodological experiment to determine if respondents who 
received an incentive to participate in the base year study (NPSAS:2000 field test) would demand 
an incentive to participate in the B&B:2000/01 follow-up study. The assessment was a follow-up 
to the incentive experiment conducted as part of the NPSAS:2000 field test. 

The NPSAS:2000 field test included an investigation of the impact on response rates of 
offering financial incentives to selected sample members, based on their survey status. Offers of 
financial incentives proved effective in enhancing response rates for particular categories of 
sample members (e.g., preliminary refusals) and was approved for use in the full-scale NPSAS 
survey. The experiment implemented as part of the B&B:2000/01 field test was designed to 
answer a series of additional questions pertaining to the B&B cohort. It was particularly 
important to determine what impact, if any, the offer of a financial incentive during the base year 
of a longitudinal survey may have on the likelihood of sample members’ response to the 
subsequent follow-up survey. Specifically: 

• What percentage of the sample members who were mailed an incentive during the 
NPSAS field test would inquire about an incentive in the B&B follow-up? 

• Among those who inquired about an incentive, what percentage would agree (or not 
agree) to do the survey in the absence of an incentive? 

• What percentage would agree (or not agree) to participate in the follow-up survey if 
initially offered the same incentive as in the base year? 

Overall, 1 83 B&B sample members were sent an incentive mailing, including $5, during 
the NPSAS field test. Of those, 132 completed the interview and received an additional $15 
incentive payment, while the remaining 51 did not complete the interview (25 were located in 
NPSAS but refused or time ran out; and the remaining 26 were not located in NPSAS). The 
B&B incentive experiment focused only on those 1 83 sample members who were mailed an 
incentive during the NPSAS field test (whether they completed the interview or not). It is 
important to decide how to handle these students in the full-scale B&B follow-up study. 

A split-sample experimental design was implemented as part of the B&B follow-up field 
test (see figure 2.3): 

(1) The 1 83 sample members who were mailed an incentive letter as part of the 
NPSAS data collection effort were stratified by respondent/nonrespondent status, 
school level, and school control. 

(2) Based upon this stratification, cases were allocated to a “control” group or an 
“experimental” group so that the two groups were identical in terms of respondent 
status, school level, and school control characteristics. A variable denoting the 
sample members’ experiment status (control or treatment) was pre loaded into the 
CATI system. 

(3) The offering of the incentive (or lack thereof) was presented to the sample 
members as follows: 



23 



B&B :2000/01 Field Test Report 



2. Design and Method of the Field Test 



• Sample members in the “experimental” or “treatment” group received an incentive 
letter with $5 cash. The letter explained the study and informed the sample members 
that they would receive a check for an additional $15 upon completion of the full 
interview. The letters were sent via express mail approximately 7 days after the 
initial student mailing was sent. 

• Sample members in the “control” group did not receive an incentive mailing (they 
did, however, receive the initial student lead letter and information leaflet sent to all 
sample members). Sample members who inquired about the incentive during an 
interview were told: “I’m sorry, but for this study we are not offering an incentive 
payment.” The interviewer then recorded whether the sample member agreed to 
continue the interview or terminated the interview. 

Results of this experiment are presented in Chapter 3. 
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Figure 2.3 — Schematic overview of B&B incentive experiment 
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Data Collection Outcomes 



A. Locating Outcomes 

1. Student locating and interviewing 

The conduct of interviews for list-based sample surveys such as B&B:2000/01 involves 
two sequential steps: locating (identifying an initial telephone number at which the sample 
member can be reached) and interviewing (convincing the sample member to cooperate and 
conducting the interview). The level of time and effort required to complete these steps with 
sample members can vary considerably. Some sample members may be reached and interviewed 
on the first attempt at contact. Others may require considerable tracing (contacting of parents, 
former roommates, etc.) before they are successfully located and interviewed. 

The time allowed for the B&B:2000/01 field test was more limited than will be the case 
in the full-scale survey. Therefore, procedures for those most difficult to locate and interview 
were constricted, with consequent adverse impact on final locating and CATI response rates. 

However, a relatively high percentage of sample members were located and interviewed 
as part of the B&B: 2000/01 field test, given the time constraints associated with conducting a 
field test. This is at least partially because the B&B:2000/01 field test is a relatively quick 
follow-up. Base-year data were collected for respondents only one year earlier, which aids the 
success of locating efforts. Even for this highly mobile population of recent college graduates, 
locating is much easier one year after initial contact than in a later follow-up. 

a. Lead letter and locator mailing to students 

One week before the start of data collection for the field test, sample members were sent 
an advance mailing that included an address update sheet. Each sample member was asked to 
review, correct, and return the sheet. Letters were mailed to 706 sample members, with 149 
cases out of a total sample of 855 unmailed because of incomplete address information. Of these 
706 sample member letters, 115 address update sheets with new or confirmed information were 
received (14 percent of the total sample). 
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b. Results of locating and interviewing effort 

Figure 3.1 presents a schematic of the outcomes of student locating and interviewing and 
related case-resolution activities. Student interview data were collected exclusively by CATI. 

No field interviewing or questionnaire mailings were conducted as part of the field test. Data 
collection for the field test lasted approximately 14 weeks, from March 28 through July 2, 2000. 

As shown in figure 3.1, attempts were made to locate 855 student sample members. 
Overall, 769 (90 percent) were located, 71 (8 percent) were not located, 6 (<1 percent) were 
considered “exclusions,” 1 and 9 (1 percent) were determined to be ineligible for the study based 
on their responses to the eligibility questions in the questionnaire. 

Student interviewing results are also shown schematically in figure 3.1. A total of 695 (of 
the 840 remaining cases after removing the exclusions) were interviewed. The majority of these 
cases (662) completed the entire interview, while 33 completed only a partial or abbreviated 
interview. A small number of the partial interviews (4) were classified as such because the 
respondent broke off after completing part of the interview. A break-off represented an explicit 
or implicit refusal or the arising of some other matter requiring the attention of the respondent, 
but such cases could not be converted or recontacted to complete the interview by the end of the 
data collection period. A substantial number (29) of these partial interviews, however, resulted 
from the administration of an abbreviated interview that consisted of a minimal set of questions 
from the full interview. Over half of this group represented interviews with Spanish-speaking 
respondents. 

Of the remaining 1 60 cases not interviewed, 7 1 were not located. A total of 74 
potentially eligible students who were located were not interviewed. Of these, 59 were explicit 
final refusals for which subsequent attempts at interviewing were determined to be infeasible or 
unwise. Not interviewed cases also included 15 sample members for whom time ran out before 
they could complete the interview; such cases clearly reflect, at least in part, the constricted data 
collection period. 2 

An overall student CATI response rate for the B&B:2000/01 field test can be calculated 
as the number of respondents interviewed divided by the initial sample size minus the 
exclusions: 

Student CATI response rate = 695/(855-1 5) = 82.7 percent. 



1 B&B exclusion cases consisted of those whose status (generally obtained through some contacted third party) was 
determined to be such that attempts at locating/interviewing them during the CATI operational period would be 
futile. The designation “exclusions” indicates that, even though the status of the case was successfully resolved, such 
cases were considered “out-of-scope” for locating and interviewing operations. Among the six B&B:2000/01 field 
test sample members classified as exclusions, five were determined to be out of the country for the duration of the 
data collection period and one sample member was verified as deceased. 

2 This group likely contained an unknown number of implicit refusal cases — i.e., individuals who after first contact 
used answering machines or ffiends/relatives as gatekeepers, as well as those who continued to make (and then 
break) appointments for an interview “in the future.” 
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Figure 3.1 — NPSAS:2000 field test result flow of locating/interviewing activities 




NOTE: Final sample including interviewed, not interviewed, and not located = 840. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 
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c. Locating and response rates for base-year respondents and nonrespondents 

Table 3.1 provides results for the B&B:2000/01 field test student locating and 
interviewing (for those located) by respondent status in the base-year study (NPSAS:2000 field 
test). 3 Some significant differences in locating and interviewing rates are evident. In terms of 
locating, just over 94 percent of the NPSAS field test respondents were located, compared to 
approximately 74 percent of NPSAS nonrespondents. There were also differences in interview 
rates among the two groups. Among base-year respondents, 93 percent of the located sample 
members successfully completed interviews during the B&B follow-up field test. The interview 
rate was nearly 30 percent lower for NPSAS:2000 field test nonrespondents, with 66 percent of 
the located sample members completing interviews. 



Table 3.1-B&B:2000/01 field test student locating and interview results, by respondent 
status in the NPSAS:2000 field test 



NPSAS:2000 respondent status 


Total 

respondents 


Located*, 


Interviewed, 


given locate 


Number 


Percent 


Number 


Percent 


Total: 


840 


769 


91.5 


695 


90.4 


NPSAS :2000 field test respondent 


725 


684 


94.3 


639 


93.4 


NPSAS :2000 field test nonrespondent 


115 


85 


73.9 


56 


65.9 



NOTE: Statistics exclude nine B&B-ineligible sample members (as determined in CAT1) and six sample members who were 
either out of the country during the field test or confirmed as deceased. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 

As shown in table 3.2, nearly one-quarter (23.6 percent) of the completed interviews for 
the B&B field test were obtained during the first week of interviewing. By the end of the third 
week of data collection, over half (5 1 .4 percent) of the interviews had been completed. After 
that, the locating and interviewing effort was much more difficult and time consuming, with the 
remaining interviews being collected over the last 1 1 weeks of data collection. 



3 The statistics in table 3.1 exclude the nine B&B:2000/01 ineligible sample members determined during CATI interviewing and 
the six exclusions; they do not exclude any potential ineligibles likely to be part of the unlocatables. 
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Table 3.2-B&B:2000/01 field test response rates, by number of weeks worked 1 



Number of weeks of data 
collection 2 


Weekly 

number 

complete 


Weekly 

percent 

complete 


Cumulative 

number 

complete 


Cumulative 

percent 

complete 


Total 


695 


100.0 


695 


100.0 


1 


164 


23.6 


164 


23.6 


2 


130 


18.7 


294 


42.3 


3 


63 


9.1 


357 


51.4 


4 


36 


5.2 


393 


56.6 


5 


58 


8.4 


451 


65.0 


6 


29 


4.2 


480 


69.2 


7 


27 


3.9 


507 


73.1 


8 


33 


4.8 


540 


77.9 


9 


11 


1.5 


551 


79.4 


10 


27 


3.9 


578 


83.3 


11 


55 


7.9 


633 


91.2 


12 


18 


2.5 


651 


93.7 


13 


28 


4.0 


679 


97.7 


14 


16 


2.3 


695 


100.0 



’Statistics in table 3.2 exclude nine B&B:2000/01 ineligible sample members and six sample members who were either out of the 
country for the duration of the field test or were deceased. 

2 Number of weeks of data collection is based on the number of weeks between the date data collection began and the final day of 
data collection. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 



d. Source of locating information for completed interviews 

Locating students in a longitudinal study to interview them is a complex task requiring 
multiple sources of information. Leads developed through one source may need to be verified 
using another data source or locating technique. 

Table 3.3 presents the original source of the telephone number at which the interview was 
completed. Tracing leads obtained via telephone during CATI data collection were the most 
important sources of these numbers, accounting for 21 1 of the final 695 completed interviews, or 
30 percent. The remaining 70 percent came from a variety of other sources, including locating 
information collected during the base-year (NPSAS:2000) study (28.4 percent), pre-data- 
collection activities using NCOA or Telematch address and telephone number processing (23.3 
percent), returns from student prenotification letters (9.2 percent), centralized tracing by TOPS 
(5.6 percent), and, finally, student call-ins to the study’s toll-free number (3.2 percent). 
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Table 3.3 — B&B:2000/01 field test source of locating information for completed interview 



Locating source . ;i|i c ^ 


Number of cases 


Percent of cases 


Total : C- ; 


695 


100.0 


New information via CATI '(|C : ^ % 


211 


30.4 


NPSAS:2000 locating information 


197 


28.4 


National Change of Address/Telematch 


162 


23.3 


Student mailing 


64 


9.2 


Intensive tracing ^ C - ' 'V ■’ ■ 


39 


5.6 


Respondent call-jn from new number ' 


22 


3.2 



NOTE: This table indicates the number and percent of cases completed, by the source that first produced the telephone number or 
address at which the interview was completed. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 



e. Student prenotification letter and address updates 

Not surprisingly, student contact and interview rates varied considerably based on 
whether or not sample members returned the address/telephone update sheet sent to them as part 
of the prenotification mailing (see table 3.4). While the return rate for these sheets was modest 
(with 1 14 of 840 eligible sample members returning update sheets, or 13 percent), the contact 
and interview rates for those who did return the forms was nearly universal. Of the 1 14 sample 
members who returned an update sheet, 113 were located by interviewers. Among those cases, 

1 10 (97.4 percent) completed the field test interview. The four who did not complete the 
interview were refusals. In short, while the percentage of update forms returned was relatively 
low, the contact and interview rates among those who did return such forms was extremely high. 



Table 3.4 — B&B:2000/01 field test locate and interview rates, by student return of address 
update form 



Student mailing status 


Total 

respondents 


Contacted 


Interviewed, given 
contact 


Number 


Percent 


Number 


Percent 


Total 


840 


769 


91.5 


695 


90.4 


Returned update form 


114 


113 


99.1 


110 


97.4 


No update form returned 


726 


656 


90.4 


585 


89.2 



NOTE: Statistics exclude nine B&B-ineligible sample members (as determined in CATI) and six sample members who were 
either out of the country during the field test or confirmed as deceased. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 



B&B:2000/01 Field Test Report 



o 

ERIC 



32 



44 



3. Data Collection Outcomes 



f. E-mail contact with sample members 

As part of the B&B:2000/01 field test effort, the use of e-mail as a means of contacting 
otherwise hard-to-reach sample members (i.e., those requiring 10 or more call attempts) was 
evaluated. E-mail addresses were collected from sample members both during the base year 
interview (NPSAS:2000) and as part of the update sheets sent to sample members. 
Approximately 8 weeks into data collection, these e-mail addresses were used to contact sample 
members who had not yet completed the B&B:2000/01 field test interview. The e-mail message 
briefly described the study, indicated our previous attempts to reach the sample member, and 
encouraged the student to contact us via telephone or e-mail to complete the survey or to 
establish a date and time for an interviewer to call. By this stage of data collection, there were 73 
sample members for whom we had a valid e-mail address (i.e., we had an e-mail address and 
when the message was sent it was not returned as “undeliverable” — the sign of a “bad” e-mail 
address). Among these, 61 (83.6 percent) were ultimately located and nearly 87 percent of those 
contacted completed the interview (see table 3.5). E-mail appears to have been an effective 
mode of communication for establishing contact with otherwise hard-to-reach sample members. 



Table 3.5 — B&B:2000/01 field test locate and interview rates, by e-mail status 



-f 7;./ v \ v ‘: • • ‘.7’ / • ' • • *• >' * * ♦ 


i f Total 


Contacted 


Interviewed, given 
contact 


E-mail status 


respondents 


Number 


Percent 


Number 


Percent 


Total 


840 


769 


91.5 


695 


90.4 


Had valid e-mail address 


73 


61 


83.6 


53 


86.9 


No e-mail address 


767 


708 


92.3 


642 


90.7 



NOTE: Statistics exclude nine B&B-ineligible sample members (as determined in CATI) and six sample members who were 
either out of the country during the field test or confirmed as deceased. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 



g. Intensive locating during data collection 

Intensive tracing efforts were required for cases in which preloaded CATI locating 
information failed to result in contact with the sample member. These intensive tracing 
activities were as follows. 

• Cases with valid addresses that were not located during the CATI operations were 
sent to FastData for telephone number updates. New information was then returned 
to CATI for further follow-up. 

• Cases returned from FastData without additional information were assigned to TOPS 
for intensive tracing. 

• Cases without valid mailing addresses were also assigned to receive intensive tracing 
from TOPS. 
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The final locate and interview rates for cases requiring centralized tracing are provided in 
table 3.6. Of the 141 cases sent to the tracing unit for intensive locating efforts, 85 were located 
(60.3 percent) and, of those located, nearly 85 percent were interviewed. In sum, although not all 
sample members were found using centralized tracing techniques, these techniques did result in 
contracts for a majority of the cases in which they were used. 



Table 3.6 — B&B: 2000/01 field test contact and interview rates, by intensive tracing efforts 





Total 


Contacted 


Interviewed, given 
contact 


Tracing status 


respondents 


Number 


Percent 


Number 


Percent 


Total 


840 


769 


91.5 


695 


90.4 


No intensive tracing required 


699 


684 


97.9 


623 


91.1 


Intensive tracing required 


141 


85 


60.3 


72 


84.7 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 

NOTE: Statistics exclude nine B&B-ineligible sample members (as determined in CATI) and six sample members who were 
either out of the country during the field test or confirmed as deceased. 

For hard-to-locate sample members, generally no single source of information is adequate 
to achieve the level of locating required. Rather, a successful locating effort requires blending 
multiple sources of information. Centralized tracing was conducted as part of the field test for 
cases in which telephone leads were exhausted during the CATI phase of data collection. 

Table 3.7 provides an overview of the sources used during intensive tracing of the hard-to-reach 
B&B:2000/01 field test sample members. Note that although the table provides information on 
the number and percent of sample members who were ultimately located when a particular 
source was used, most of the cases were traced using multiple sources. 



Table 3.7 — B&B:2000/01 field test locate rates, by tracing source used during intensive 
tracing efforts 



Tracing source 


Cases involving intensive tracing 


Total 


Respondents located 


Number 


Percent 


Consumer database search 


137 


83 


60.6 


Directory assistance 


118 


69 


58.5 


Database - address search 


82 


45 


54.9 


Database - reverse phone lookup 


57 


31 


54.4 


Database - name search 


66 


31 


47.0 


Internet search 


53 


23 


43.4 


Database - neighbor search 


18 


4 


22.2 



NOTE: Most cases were traced using multiple sources so numbers and percentages add up to more than the total. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 
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Contact was made with sample members in 61 percent of the cases where information 
from consumer databases was used as part of the locating effort. Directory assistance was a 
factor in locating about the same percentage of hard-to-locate sample members (58.5 percent). 
Over half of the cases for which address search databases (54.9 percent) or reverse telephone 
look-ups (54.4 percent) were used resulted in contact with a sample member. 

If more extensive searches were required (such as generic database name searches or 
Internet searches), the percentage of sample members located was reduced. These techniques 
were used only if previous search efforts failed to provide sufficient locating information. Fewer 
than half of those cases for whom these more extensive tracing techniques were required were 
ultimately located (47 percent for name searches and 43.4 percent for Internet searches). Finally, 
just one in five (22.2 percent) of those for whom neighbor searches were required were located. 4 

2. Refusal conversion efforts 

Refusal conversion procedures were used to gain cooperation from individuals who 
refused to participate when contacted by telephone interviewers. Refusals came not only from 
sample members, but also from spouses, housemates, parents, and other gatekeepers. When 
either a sample member or a gatekeeper refused to participate in the locating or interviewing 
effort, the case was referred to a specially trained refusal-conversion specialist in the Telephone 
Survey Unit. There were 195 initial refusals among the field test sample (23 percent of the 
initially fielded sample of 855). Most refusals came from sample members (122 refusals), 
although 73 refusals were by other contacted individuals (see table 3.8). In all, 59 percent of the 
cases with initial refusals were successfully converted into completed interviews. The 
conversion rate was 52 percent among refusing sample members. 



Table 3.8 — B&B:2000/01 field test conversion of initial refusals, by source of refusal 



Who refused 


Number of initial 
refusals 


Respondents interviewed, given initial refusal 


Number 


Percent 


Total 


195 


114 


58.5 


Sample member 


122 


63 


51.6 


Other individual 


73 


51 


69.9 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 



The success of converting refusals varied according to the sample member’s response 
status in the base-year study (see table 3.9). Among respondents to the NPSAS:2000 field test 
interview, 68 percent of the sample members who initially refused to be interviewed (or whose 
gatekeeper refused) ultimately completed the B&B:2000/01 field test interview. In contrast, only 
26 percent of the base-year nonrespondents were successfully converted. 



4 Neighbor searches involve the use of databases to identify the addresses and telephone numbers of properties or 
apartments located in close proximity to the sample member’s last known address. The assumption is that these are 
current or former neighbors who may be able to provide current locating information for the sample member. 
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Table 3.9 — B&B:2000/01 field test conversion of initial refusals, by respondent status in 
NPSAS:2000 field test 



NPSAS:2000 field test 
respondent status 


Number of initial 
refusals 


Respondents interviewed, given initial refusal 


Number 


Percent 


Total 


195 


114 


58.5 


Respondent 


152 


103 


67.8 


Nonrespondent 


53 


11 


25.6 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
S tudy : 2000/0 1 fiel d test . 



3. Reliability reinterview 

A subsample of eligible sample members who completed the B&B:2000/01 field test 
interview was selected to participate in a reliability reinterview, containing a subset of items from 
the initial interview. A random selection algorithm was programmed directly into the CATI 
instrument. Sample members selected for the reinterview were informed of their selection at the 
end of the initial interview and allowed an opportunity to agree to the reinterview or to refuse it 
at that time. 

A total of 83 respondents were selected for the reliability reinterview. Due to the built-in 
delay in administering the reinterview (a delay of approximately 3-4 weeks from the initial 
interview) and the need to complete reinterviews during the same time frame as the field test 
interview, those selected for reinterview were more likely to be those sampled and interviewed 
early during the data collection period for the field test. Such individuals were those most easily 
located and convinced to participate in the initial interview. Consequently, the reported 
agreement and reinterview rates are probably higher than if the reinterview respondents had been 
sampled subsequent to the initial data collection effort. 

4. Interview burden and effort 

This section of the field test report reviews the effort and burden associated with the 
B&B:2000/01 field test student interview. We examine the interview’s length by considering the 
timing analysis statistics. This information is useful because it provides evidence that can reduce 
respondent burden, reduce data collection effort and cost, and improve data quality. Then we 
consider the effort required to locate and interview sample members for the study using the 
average interview time. 

During CATI instrument development, project staff embedded time stamps at the start 
and end of the interview, as well as the beginning and end of each interview screen, which could 
include up to eight related items. The time stamps measured the elapsed time to complete each 
segment of the interview, and enabled project staff to monitor the time required to complete 
specific interview items, the online coding programs, sections of the interview, and the entire 
interview. 
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The time (in minutes) needed to conduct a student interview is shown by interview 
section in table 3.10. Sections are listed in the table in the order in which they were presented. 
Certain sections of the interview applied to selected groups of respondents (see figure 2.2,) so 
timing results are presented for the overall cohort, and by subgroup. For example, Section A was 
designed for base-year nonrespondents, so the number of cases in that group was less than for the 
rest of the instrument. Respondents who were currently teaching skipped the post-baccalaureate 
employment section and proceeded directly to the teaching section. 

Table 3.10 presents timing results for the B&B:2000/01 field test cohort. Overall average 
administration time to complete the student interview was 18 minutes. There was no difference 
in average completion time due to base-year response status (see table 3.1 1). Both respondents 
and nonrespondents to the NPSAS:2000 field test took an average of 18 minutes to complete the 
interview. 5 For respondents who had taught since graduating (see table 3.12), the average 
interview time was 21 minutes compared to 17 minutes for those who had not taught. 

The Technical Review Panel reviewed the administration time and then recommended 
certain items for deletion in the full-scale study. Items to be excluded typically showed a lack of 
temporal stability or extremely low variance of responses (see chapter 5). 

Interview administration time, however, reflected only a small fraction of the time 
required to obtain a completed interview. Time was spent by locator/interviewers in locating, 
scheduling call-backs, attempting refusal conversion, and other related activities. This time was 
spent whether or not interviews were obtained. The average locator/interviewer time 
requirement for each completed interview was slightly more than 2 hours. 

Table 3.10 — Average minutes to complete B&B:2000/01 field test student interview, by 



interview section 



CATI section 1 


Average time 


Number of cases 


Section A - eligibility determination for NPSAS non-respondents 


0.48 


50 


Section B - enrollment history 


5.38 


665 


Section D - student background 


2.93 


663 


Section E - post-baccalaureate education 


2.64 


665 


Section F - employment/income 


4.66 


663 


Section G - teaching experiences 


2.40 


662 


Total interview 


18.22 


666 



1 All the original Section C items were moved to other sections. To avoid introducing confusion into the CATI programming, 
however, the remaining sections have not been relettered. 

NOTE: A section was considered complete if the amount of time to complete the section was greater than zero and the section 
completion flag was set. Section outliers were removed from the timing calculations (2 in section A, 1 in section B, 3 in 
section D, 1 in section F, and 2 in section G). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study :2000/01 field test. 



5 Although base-year non respondents had to complete an extra section (to determine study eligibility,) the average 
time was actually shorter than for base-year respondents. However, the eligibility determination items were very 
short, and the number of cases who went through the eligibility determination section was very small relative to the 
total number of respondents. 
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Table 3.11 — Average minutes to complete B&B:2000/01 field test student interview, by 
interview section and NPSAS:2000 response status 





Total 


NPSAS:2000 field test 
nonrespondent 


NPSAS:2000 field test 
respondent 


CATI section 1 


Average 

time 


Number of 
cases 


Average 

time 


Number of 
cases 


Average 

time 


Number of 
cases 


Section A - eligibility determination for NPS AS 
nonrespondents 


0.48 


50 


0.48 


50 


( ) 


0 


Section B - enrollment history 


5.38 


665 


5.50 


52 


5.37 


613 


Section D - student background 


2.93 


663 


2.73 


51 


2.95 


612 


Section E - post-baccalaureate education 


2.64 


665 


2.09 


52 


2.69 


613 


Section F - employment/income 


4.66 


663 


4.80 


52 


4.65 


611 


Section G - teaching experiences 


2.40 


662 


1.99 


52 


2.44 


610 


Total interview 


18.22 


666 


17.84 


52 


18.26 


614 



t Not applicable. 

l All the original Section C items were moved to other sections. To avoid introducing confusion into the CATI programming, 
however, the remaining sections have not been relettered. 

NOTE: A section was considered complete if the amount of time to complete the section was greater than zero and the section 
completion flag was set. Section outliers were removed from the timing calculations (2 in section A, 1 in section B, 3 in 
section D, 1 in section F, and 2 in section G). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 



Table 3.12 — Average minutes to complete B&B:2000/01 field test student interview, by interview 
section and teaching status 





Total 


B&B:2000/01 field test 
nonteachers 


B&B:2000/01 field test 
teachers 


CATI section 1 


Average 

time 


Number of 
cases 


Average 

time 


Number of 
cases 


Average 

time 


Number of 
cases 


Section A - eligibility determination for NPSAS 
nonrespondents 


0.48 


50 


0.46 


40 


0.59 


10 


Section B - enrollment history 


5.38 


665 


5.25 


504 


5.80 


161 


Student background 


2.93 


663 


2.88 


504 


3.08 


159 


Section E - post- baccalaureate education 


2.64 


665 


2.58 


504 


2.84 


161 


Section F - employment/income 


4.66 


663 


5.78 


502 


1.19 


161 


Section G- teaching experiences 


2.40 


662 


0.80 


502 


7.44 


160 


Total interview 


18.22 


666 


17.47 


505 


20.58 


161 



1 All the original Section C items were moved to other sections. To avoid introducing confusion into the CATI programming, 
however, the remaining sections have not been relettered. 

NOTE: A section was considered complete if the amount of time to complete the section was greater than zero and the section 
completion flag was set. Section outliers were removed from the timing calculations (2 in section A, 1 in section B, 3 in 
section D, 1 in section F, and 2 in section G). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 



38 



o 

ERIC 



50 



B&B .2000/01 Field Test Report 



3. Data Collection Outcomes 



5. Interviewer hours 

During the B&B:2000/01 field test, telephone interviewers worked a total of 1,374 hours 
to obtain completed interviews from 695 sample members who completed full or partial 
interviews and 79 individuals who completed reliability reinterviews in CATI. Excluding the 
time each interviewer spent in training or attending quality circle meetings, and the additional 
time that team leaders and other senior telephone interviewers allotted to supervision and 
monitoring, this represented an average of 2.0 hours per completed interview. Since the average 
time to administer the interview was just over 1 8 minutes, most interviewer time clearly was 
spent in other activities, primarily in locating and contacting. In addition to the telephone 
interviewers, supervisors and monitors worked 488 hours during the field test, or approximately 
1 hour for every 3 hours of telephone interviewing. Another 202 hours were attributed to refusal- 
conversion training, quality circle meetings, and debriefing meetings. 

The level of effort required to complete interviews varied considerably across shifts and 
days of the week. As shown in table 3.13, the lowest “hours per complete interview” average 
was obtained on Mondays, particularly during the evening shift (average of 1 .5 hours per 
complete). The highest hours per complete were incurred on Thursdays during the day shift 
(average of 2.7 hours per complete). The relatively small size of the interviewing staff on a 
particular shift (ranging from two to five interviewers per shift) makes it difficult to determine 
whether these variations are due to the availability (or lack thereof) of sample members during 
those periods or to variations in interviewer efforts on those shifts. 



Table 3.13 — B&B:2000/01 field test average hours per complete, by day of the week and 
time of day 



Time of shifts 


Day of the week l 


Overall 

average 


Monday 


Tuesday 


Wednesday 


Thursday 


Friday 


Saturday 1 


Sunday 2 


Total 


2.0 


1.5 


1.9 


2.0 


2.1 


2.2 


2.2 


2.2 


8:30 a.m. -4:59 p.m. 


2.1 


1.7 


1.9 


2.5 


2.7 


1.9 


(t) 


(t) 


5:00 p.m. -11:59 p.m. 


1.9 


1.5 


1.9 


1.8 


2.0 


2.3 


(t) 


(t) 



| No data available for this time slot given the different schedule on weekends. 

1 Interviewing hours for Saturday were 9:00 am-5:00 p.m. 

2 Interviewing hours for Sunday were 1 :30 p.m -9:30 p.m. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 
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6. Number of Calls 

Telephone interviewers made 1 5,347 telephone calls during the field test, with an average 
of 1 7.9 calls per sample member. 6 An average of 1 5.4 calls was required to obtain the 695 
completed cases. Slightly more than half of the completed telephone interviews (52.5 percent) 
were completed with fewer than 10 calls, 32.0 percent required 10 to 29 calls, and 15.5 percent of 
the completed cases required 30 or more call attempts. 

Of the 1 5,347 calls made, 1 in 5 resulted in contact with an individual. Nearly half of the 
contacts (47.1 percent or 7,234 calls) resulted in contact with an answering machine, reflecting 
the heavy usage of such devices among this population. An answering machine was reached at 
least once (and often multiple times) for three of every four B&B sample members. The 
remaining one-third (33.7 percent) of the calls made were other noncontacts (busy, ring/no- 
answer, fax line, pager, etc.). 

7. Answering machines, messages, and call-ins 

Answering machines are an increasing problem for all studies conducted by telephone. 
Whether the machines are used to screen unwanted calls or used to facilitate “on the go” 
lifestyles, answering machines pose an obstacle to contacting sample members and completing 
interviews. On average, the higher the percentage of calls resulting in an answering machine 
disposition, the greater the average number of call attempts required to complete the interview. 
Where no machine was encountered, an average of 6.0 calls was required to obtain a completed 
interview. If fewer than 50 percent of the call attempts reached an answering machine, it took an 
average of 14.5 call attempts to complete the interview. Finally, among cases where an 
answering machine was reached on 50 percent or more of the call attempts, it took on average 
24.9 call attempts to complete an interview. 

Answering machines are not, however, insurmountable barriers. Table 3.14 provides the 
contact and interview rates for hard-to-reach cases. As the table shows, ultimately we were able 
to contact over 90 percent of those hard-to-reach sample members where an answering machine 
was encountered on one or more call attempts. This high success rate holds even for cases where 
a machine was encountered on 50 percent or more of the call attempts. Likewise, completed 
interviews were obtained from 85 percent of these hard-to-reach cases, despite the interviewer 
reaching an answering machine on one or more attempts. 



6 These figures are based on calls made by telephone interviewers and exclude calls made by TOPS in the course of 
attempting to locate sample members. 
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Table 3.14 — B&B:2000/01 field test contact and interview rates for hard-to-reach 

respondents, by percentage of calls where an answering machine was reached 



Percent of calls resulting in 
answering machine 


Totalis i 
respondents* 


; f Contacted 


Interviewed, 


given contact 


Nuinlxer 


Percent 


Number 


% Percent 


1 Total 


434 


387 


89.2 


330 


85.3 


$0% 


39 


27 


69.2 


24 


88.9 


Lessthan 50% 


214 


194 


90.7 


165 


85.1 


50% or more 


181 


166 


91.7 


141 


84.9 



^Calculations include only cases with 10 or more call attempts (i.e., those considered to be hard to reach). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 



Answering machines can also serve as a vehicle for making contact with a sample 
member. Messages left on answering machines are the functional equivalent of “electronic lead 
letters,” notifying sample members of an impending call from an interviewer. During the B&B 
field test, the following message was left the first and fourth time an answering machine was 
encountered at a particular telephone number: 

“I am calling for the U.S. Department of Education about a research study in 
which (sample member name) has been selected to participate. Please ask 
(him/her) to call “(name)” at 1-800-555-5555, and refer to ID number 
"(unique ID)" to complete the study. Thank you.” 

The message (1) notified the sample members that they had been selected for a research 
study and (implicitly) that they would be recontacted in the near future, and (2) encouraged them 
to call in to complete the interview. As shown in table 3.15, a sizable portion of the field test 
sample did call in to complete the interview. In total, 165 callers used the toll-free number 
established for the study. Among these, 86.1 percent (142 cases) completed the interview upon 
calling in. Among those who did not complete the interview when they called in, the calls were a 
relatively even mix of refusals by the sample member, contact persons calling to provide new 
locating information for the sample member, or contacted individuals calling to say they did not 
know the sample member or did not know where to contact him or her. 
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Table 3.15 — B&B:2000/01 field test interview results, by call-ins to toll-free study number 



Message left on answering machine fe 


Total number of call-ins 


Respondents interviewed 


Ui: Number 


- Percent 


Total 


165 


142 


86.1 


No message 


18 


14 


77.8 


Message left 


147 


128 


87.1 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 

Note: Because respondents were not asked directly what encouraged them to call in, it is not possible to examine directly the 
impact on call-ins of leaving messages versus obtaining the toll-free number from some other source, such as a third-party 
contact, initial student lead letter, or follow-up incentive mailing. The results presented here, therefore, simply provide a 
breakout of call-ins by respondents who received a message on their answering machine and those who did not, because no 
answering machine was encountered on any attempt to reach the respondent. 



B. B&B:2000/01 field test incentive experiment 

1. Overview of the incentive experiment 

As explained in section 2.H.1, the incentive experiment implemented during the 
B&B:2000/01 field test was designed to examine the likelihood that sample members in the 
follow-up study who received an incentive payment in the base-year study would respond to a 
request for a follow-up study. In particular, the questions of interest involved (1) whether or not 
sample members who received a modest incentive ($20) for their participation in the 
NPSAS:2000 field test study would participate in the follow-up without an incentive, and (2) if 
such respondents received an incentive at the beginning of the follow-up study, whether they 
would respond more quickly than those not receiving an incentive, thereby lowering the level of 
effort for such cases. 

NPSAS:2000 sample members who received an incentive in the base-year study were 
randomly assigned to one of two groups for the B&B field test: an incentive group and a control 
group. Incentive group members received a $20 incentive the first week of data collection to 
complete the interview; control group members received no incentive. All other survey 
activities, such as contacting/interviewing, tracing/locating efforts, and the like, were similar for 
the two groups. Sample members selected to receive an incentive were sent a personalized letter 
delivered by express overnight service. Enclosed with the letter was a $5 bill and instructions for 
completing the interview by calling a toll-free telephone number. After successfully completing 
the B&B: 2000/01 interview, whether by call-in to the toll-free number from the sample member 
or through a subsequent call from a telephone interviewer, each respondent in the treatment 
group received an additional payment of $15 by personalized check. 

2. Results of the incentive experiment 

Of the 179 cases in the field test who were eligible for the experiment, 90 cases were 
selected to receive an incentive, and 89 cases were assigned to a control group. As shown on 
table 3.16, 60 percent (54 of the 90 cases) of the sample members who received an incentive 
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completed the interview during the time frame in which the experiment was conducted. Among 
the control group, completed interviews were obtained by half of the sample members (45 of 89 
cases). The difference between those who received an incentive and those who did not is not 
statistically significant. 

In terms of level of effort (see table 3.17), it took fewer call attempts to obtain completed 
interviews with sample members who received the incentive mailing at the outset of the study 
compared to those in the control group (9.6 versus 12.2 call attempts). These results, however, 
are not statistically significant either. 

Finally, the experiment was also designed to allow interviewers to note whether or not 
sample members — particularly those in the control group — inquired about the incentive at the 
outset of the study. Among respondents who did not receive an incentive mailing, only six asked 
about the possibility of receiving an incentive for their participation in the B&B:2000/01 follow- 
up: two refused to participate and four continued to participate upon learning that they would not 
be eligible for the incentive. 



Table 3.16 — B&B:2000/01 field test interview outcome, by incentive experiment group 





Total number in 
group 


Interview outcome 


Incentive experiment group 


Number complete 


Percent complete 


No incentive 


89 


45 


50.6 


Received incentive 


90 


54 


60.0 



NOTE: Significance: p < .05 using a Chi-square test of significance. This analysis showed no significant difference between 
incentive groups. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study: 2000/01 field test. 



Table 3.17 — B&B:2000/01 field test mean call attempts for completed interviews, by 
incentive experiment group 





Interview outcome 


Incentive experiment group 


Number complete 


Mean call attempts 
per complete 


No incentive 


45 


12.2 


Received incentive 


54 


9.6 



NOTE: Significance: p < .05 using an F-test of means. This analysis showed no significant difference between incentive 
groups. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Baccalaureate and Beyond Longitudinal 
Study:2000/01 field test. 
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